Image processing apparatus and method, and image sensing apparatus

ABSTRACT

An image processing method is for processing image data that has been captured by an image sensing apparatus having an image sensing device in which a plurality of photoelectric conversion elements are arrayed two-dimensionally for outputting an electric signal in accordance with an incident amount of light. According to the method, attitude information representing the attitude of the image sensing apparatus is input and, in a case where the input attitude information of the image sensing apparatus indicates an attitude that has been specified in advance, the image data is subjected to rotation processing, which conforms to the attitude specified in advance, successively in units of a prescribed number of pixels divided into a plurality of areas.

FIELD OF THE INVENTION

This invention relates to an image processing apparatus and method aswell as an image sensing apparatus. More particularly, the inventionrelates to an image processing apparatus and method and an image sensingapparatus for applying processing, which conforms to the attitude of theimage sensing apparatus, to an image captured by the image sensingapparatus.

BACKGROUND OF THE INVENTION

Conventionally, an image processing apparatus for processing an imagecaptured by an image sensing apparatus such as a digital camera recordsthe image data representing the captured image in a recording unitfollowing compression or as is if compression has not been applied.Further, the image processing apparatus reads the image data out of therecording unit at the time of playback and displays the image on thedisplay screen of a monitor, irrespective of the attitude of the cameraat the time of image sensing, following expansion if the data has beencompressed or as is if the data has not been compressed.

FIGS. 10A to 10C illustrate the relationship between the attitude of acamera when an image is taken and the image displayed on the monitor.FIG. 10A shows the observed image that appears in an image sensing areaviewed from the finder of the camera at the time of image sensing. Thisillustrates a situation where an image is taken with a camera attitudein which the image sensing area is long in the width direction (this isreferred to as “landscape orientation”). FIG. 10B illustratesconceptually the image data that is output from an image sensing devicewhen an image is taken in the landscape orientation shown in FIG. 10A.FIG. 10C illustrates this image when it has been displayed on a displayunit such as a TV. In this case it is understood that the direction ofthe image displayed is the same as the direction of the observed imageas viewed from the finder at the time of image sensing.

FIGS. 11A to 11C similarly illustrate the relationship between theattitude of the camera when an image is taken and the image displayed onthe monitor. FIG. 11A shows the observed image that appears in the imagesensing area viewed from the finder of the camera at the time of imagesensing. This illustrates a situation where an image is taken with acamera attitude in which the image sensing area is long in the heightdirection (referred to as “portrait orientation”). In this example, thecamera has been rotated 90° in the clockwise direction. FIG. 11Billustrates conceptually the image data that is output from the imagesensing device when an image is taken in the portrait orientation shownin FIG. 11A. FIG. 11C illustrates this image when it has been displayedon a display unit such as a TV. In this case the image displayed on thedisplay unit has been rotated by 90° in the counter-clockwise directionwith respect to the vertical direction of the observed image viewed fromthe finder by the user at the time of image sensing.

If FIGS. 10C and 11C are compared, it will be appreciated that thedirection of the displayed image is correct in FIG. 10C for the casewhere the image was taken in the landscape orientation. As a result, onecan enjoy the image as is without experiencing an odd sensation.However, in FIG. 11C where the image was taken in the portraitorientation, the direction, in which the image has been rotated 90° inthe counter-clockwise direction, of the displayed image is wrong. Thisimage cannot be enjoyed as is without experiencing an odd sensation.

In order to deal with this problem, the specification of Japanese PatentApplication Laid-Open No. 10-233993 proposes detecting the attitude ofthe camera at the time of image sensing using a camera attitude sensorand, in accordance with the detected camera attitude, changing the orderin which image data, which has been stored in a frame memory, is readout, thereby generating compressed image data representing an image inwhich the direction of the observed image at the time of image sensingbecomes the same as the direction of the image displayed on a displayunit.

Further, the specification of Japanese Patent Application Laid-Open No.10-336660 proposes providing a camera orientation detector for detectingwhether the longitudinal direction in an image sensing area of thecamera is the portrait orientation or the landscape orientation, and twotypes of quantization tables, namely one for the portrait orientationand one for the landscape orientation. A quantization table selectorselects one of the quantization tables in accordance with the cameraorientation and outputs the selected table to a quantization processingcircuit. Further, DCT (Direct Cosine Transform) coefficients that havebeen output from a DCT processing circuit are quantized in thequantization processing circuit using the quantization table selected.

However, in a case where processing for rotation by 90° in the clockwisedirection or 270° in the clockwise direction (namely by 90° in thecounter-clockwise direction) at the time of display is executed inaccordance with attitude information, which prevailed at the time ofimage sensing, that has been appended to the image data, so that animage captured in the portrait orientation can be enjoyed without an oddsensation, the rotation processing takes a considerable time. Since thetime required by this rotation processing lengthens as the imageinformation, namely the number of pixels, increases, there is anincrease in waiting time from the moment a switch member for reproducingand displaying an image is operated to the moment the image isdisplayed.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation and its object is to make it possible to eliminate a temporarystorage memory (work register) for rotation processing, shorten the timeit takes to execute processing for rotating an image that has beencaptured in the portrait orientation and reproduce the image in adisplay in the portrait orientation on a display unit in a playbackprocessing time equal to that of an image captured in the landscapeorientation.

According to the present invention, the foregoing object is attained byproviding an image processing apparatus for processing image data thathas been captured by an image sensing apparatus having a two-dimensionalimage sensing device, the apparatus comprising: an attitude informationinput unit that inputs attitude information representing the attitude ofthe image sensing apparatus; and a rotating unit which, in a case wherethe attitude information of the image sensing apparatus that has beeninput by the attitude information input unit indicates an attitude thathas been specified in advance, subjects the image data to rotationprocessing, which conforms to the attitude specified in advance,successively in units of a prescribed number of pixels corresponding todivided areas of the image data.

According to another aspect of the present invention, the foregoingobject is attained by providing an image sensing apparatus having atwo-dimensional image sensing device, comprising: an attitude detectingunit that detects the attitude of the image sensing apparatus; and arotating unit which, in a case where the attitude of the image sensingapparatus that has been detected by the attitude detecting unitindicates an attitude that has been specified in advance, subjects theimage data to rotation processing, which conforms to the attitudespecified in advance, successively in units of a prescribed number ofpixels corresponding to divided areas of the image data.

In still another aspect of the present invention, the foregoing objectis attained by providing an image processing method for processing imagedata that has been captured by an image sensing apparatus havingtwo-dimensional image sensing device, the method comprising: inputtingattitude information representing the attitude of the image sensingapparatus; and in a case where the attitude information of the imagesensing apparatus indicates an attitude that has been specified inadvance, subjecting the image data to rotation processing, whichconforms to the attitude specified in advance, successively in units ofa prescribed number of pixels corresponding to divided areas of theimage data.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram illustrating the structure of a digital cameraaccording to a first embodiment of the present invention;

FIGS. 2A to 2D are diagrams useful in describing read-out processingexecuted when an image is taken in the landscape orientation accordingto the first embodiment;

FIGS. 3A to 3E are diagrams useful in describing read-out processingthat involves a rotation processing when an image is taken in theportrait orientation according to the first embodiment;

FIG. 4 is a diagram useful in describing control for read-out to a workregister when an image is taken in the portrait orientation according tothe first embodiment;

FIGS. 5A to 5C are diagrams useful in describing a rotation processingaccording to the first embodiment;

FIGS. 6 is a diagram useful in describing a rotation processingaccording to the first embodiment;

FIGS. 7A to 7C are diagrams illustrating the relationship among anobserved image when an image is taken in the portrait orientation, theimage data and the image reproduced and displayed according to the firstembodiment;

FIG. 8 is a block diagram illustrating the structure of a digital cameraaccording to a second embodiment of the present invention;

FIG. 9 is a block diagram illustrating the structure of a digital cameraaccording to a third embodiment of the present invention;

FIGS. 10A to 10C are diagrams illustrating the relationship among anobserved image when the image is taken in the landscape orientation, theimage data and the image reproduced and displayed according to the priorart; and

FIGS. 11A to 11C are diagrams illustrating the relationship among anobserved image when the image is taken in the portrait orientation, theimage data and the image reproduced and displayed according to the priorart.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

A first embodiment of the present invention will now be described.

FIG. 1 is a block diagram illustrating the structure of an image sensingapparatus such as a digital camera according to a first embodiment ofthe present invention.

Depicted in FIG. 1 are an image sensing unit 13 comprising a CCD areasensor or the like; an original image processor 141 for processing animage signal from the image sensing unit 13; a first memory 142; arotation processor 7; an encoder 10; a second memory 143; a storage unit9 comprising a non-volatile memory having enough capacity to hold imagedata of a plurality of images; an attitude detector 8 for detecting theattitude of an image processing apparatus; a microcontroller (PRS) 140for controlling overall image processing; a display memory 144 and adisplay unit 6 such as an LCD. Switches 101, 102, 103 and 104 are alsoprovided for changing over input and output of data, as will bedescribed later. The PRS 140 is a one-chip computer provided internallywith a CPU (Central Processing Unit), RAM, ROM, EEPROM (ElectricallyErasable Programmable ROM) and input/output ports, etc. The PRS 140executes a series of operations based upon a sequence program that hasbeen stored in the ROM.

The characterizing feature of the present invention resides in therotation processor 7, described later. In FIG. 1, therefore, an arrowsymbol from the PRS 140 has been attached only to the rotation processor7 for the purpose of making it clear that it is under the control of thePRS 140. However, the other components in FIG. 1 also are under thecontrol of the PRS 140.

The image sensing unit 13 comprises an image sensing device such as aCCD area sensor and a sensor driver for driving the image sensingdevice. When light from a subject forms an image on the image sensingdevice through an optical system (not shown), the image sensing devicesubjects the light to an photoelectric conversion and outputs theresultant electric signal to the original image processor 141. Theoriginal image processor 141, which has an A/D conversion function,converts the analog electric signal from the image sensing unit 13 todigital data, reduces noise components by a low-pass filter, executes aseries of image processing operations relating to so-called imagecomposing, such as pixel and color interpolation processing as well as awhite balance correction and gamma correction, and separately outputscolor components R, G and B.

The signals output from the original image processor 141 are stored inthe first memory 142 on an RGB color-component basis by operation of theswitch 101. They are also sent to the display memory 144 for display onthe display unit 6, such as an LCD (Liquid Crystal Display, with whichthe camera is equipped.

When one screen of image data is stored in the first memory 142 as thecolor components R, G, and B, the image data is sent to the rotationprocessor 7 successively on a per-RGB component basis via the switch102. The rotation processor 7 includes a work register A71, a datastring converter 72, a work register B73 and a switch 74 for selectingand outputting the output of the work register A71 or work register B73.After the image data has been separated into the R, G and B colorcomponents, each color component of the image data that has been storedin the first memory 142 is stored in the work register A71 and workregister B73. The work register A71 and work register B73 each have acapacity for storing 64 pixels of any color component.

The rotation processor 7 processes the image data on aper-color-component basis in accordance with the attitude of the digitalcamera when an image is taken and outputs the processed data to theencoder 10. The determination as to whether the digital camera is in thelandscape orientation or portrait orientation is made by the attitudedetector 8. The PRS 140 receives the output of the attitude detector 8and changes over the nature of control in the rotation processor 7.Specifically, when an image has been taken in the landscape orientation,the image data that has entered the work register A71 via the switch 102is output to the encoder 10 via the switch 74. On the other hand, whenan image has been taken in the portrait orientation, the image data thathas entered the work register A71 via the switch 102 is directed throughthe data string converter 72 and work register B73 and is then output tothe encoder 10 via the switch 74 upon being subjected to processingdescribed later.

A DCT (Discrete Cosine Transform) and a Huffman conversion, which arebased on the JPEG format, are executed as examples of encoding andcompressing processing performed by the encoder 10. Since a DCT andHuffman conversion per se are well known, these will not be describedhere. It should be noted that the present invention is not limited bythe method of encoding and compression and that it is possible to useother well-known methods. Besides a DCT and Huffman transform,processing that conforms to the encoding and compressing method, such asa discrete wavelet transform, may be executed as appropriate.

After encoding and compression have been executed by the encoder 10, theimage data of each color is stored temporarily in the second memory 143on a per-color basis via the switch 103. The above-described imageread-out from the first memory 142, processing by the rotation processor7 and encoding processing by the encoder 10 are repeated first for onescreen's worth of the R component, and encoding processing is executedone screen at a time on a per-color basis, namely for one screen's worthof the G component and one screen's worth of the B component. Before orafter this processing, the PRS 140 generates a header and/or footer(information such as the date and time of image sensing operation) ofany format with regard to the encoded image data as necessary andrecords the header and/or footer in the second memory 143 together withthe image data.

The image data of each color and the header and/or footer of any formatthat have been recorded in the second memory 143 are output to thestorage unit 9 via the switch 104.

The storage unit 9, which is a non-volatile storage member having enoughcapacity to hold image data of a plurality of images, is removablyloaded in the image sensing apparatus itself. Accordingly, with thestorage unit 9 loaded in the image sensing apparatus, after image datafrom the second memory 143 is stored in the storage unit 9 via theswitch 104 for a plurality of images, the storage unit 9 is thenceforthremoved from the image sensing apparatus and can be loaded in anothersystem or apparatus that is capable of reading the image data in a dataformat the same as that of the present image sensing apparatus, and thestored image data can be reproduced, edited or saved in this othersystem or apparatus.

Next, the rotation processing operation will be described with referenceto FIGS. 2A to 6. Described first will be an instance where an image hasbeen taken in the landscape orientation, in which case rotationprocessing is not executed.

FIGS. 2A to 2D illustrate processing in a case where an image was takenwith the camera being held in the landscape orientation. These drawingsillustrate the relationship among the area of a subject at the time ofimage sensing, the output image of the image sensing unit 13 in FIG. 1and the data stored in the first memory 142 and second memory 143. Herea case will be described where a plane in which a plurality of lettershave been written is adopted as the subject area so as to clarify thedifference in nature of signal processing that is based upon cameraattitude at the time of image sensing.

In FIG. 2A, a zone enclosed by the dashed line indicates a subject areain which letter information “A” to “S” and “a” to “s” comprises acertain plane. Further, a zone enclosed by the solid line is anobservation area as seen from the camera finder. This area coincideswith the image sensing area. Here the letter information “A” to “H” and“l” to “s” lies outside the image sensing area and is not imaged. FIG.2B is a conceptual view of image data that is output from the imagesensing device, and it will be understood that the character information“A” to “H” and “l” to “n” is not being imaged.

The image data representing the sensed image is separated into the R, G,and B color components by a series of processing operations in theoriginal image processor 141 in the order in which the image data isread out of the image sensing device, and the data is stored in thefirst memory 142 on a per-color-component basis.

FIG. 2C illustrates the order in which 8×8 pixels, which serve as oneunit, are read out of the first memory 142. If an image has been takenin the landscape orientation, the order of read-out from the firstmemory 142 in terms of the imaged areas of 8×8 pixels is “I” to “P”, “Q”to “S” to “a” to “c” and “d” to “k”, as indicated by the arrows. Thus,in units of 8×8 pixels, a raster scan is converted to a zigzag scan toread image data out of the first memory 142. The reason for adopting 8×8pixels as the unit of read-out from the image sensing device involvesthe interface between the encoder 10 and rotation processing, and it isparticularly preferred in view of the processing speed that read-out isto be performed in the processing units of encoding and compression inencoder 10.

After the image data that has been read out of the first memory 142 istransferred to the work register A71 within the rotation processor 7,the image data is sent to the encoder 10 via the switch 74 and issubjected to encoding processing, after which the encoded data is storedin the second memory 143. At this time the switch 74 selects the “0”side to thereby select the output data from the work register A71.

FIG. 2D illustrates the status of the image data at the moment ofstorage in the second memory 143 in a case where read-out has beenperformed in the order described above.

As illustrated in FIG. 2, if the image was taken with the camera in thelandscape orientation, “k”, for example, is at the lower right cornerboth when it is stored in the first memory 142 and in the second memory143. Likewise, “I” is at the upper left. Thus, there is no change in therelative positional relationship of the plurality of blocks, eachcomprising 8×8 pixels, in the memory. It should be noted that since datathat has undergone encoding processing is stored in the second memory143, FIG. 2D of this stored data is a conceptual view regarding thecontent of the second memory 143 illustrated so that it is easy tocompare with the content of the first memory 142.

The processing up to read-out of the image signal from the first memory142 and storage in the second memory 143 is executed in order, onescreen at a time, with regard to each of the color components R, G andB.

Thus, in a case where an image has been taken in the landscapeorientation, the work register A71 performs the role of raising theprocessing speed relating to encoding processing as a function of atemporary storage memory when data is sent from the first memory 142 tothe encoder 10.

Next, reference will be had to FIGS. 3A to 3E to describe processing ina case where an image has been taken with the camera in the portraitorientation. In FIGS. 3A to 3E, it is assumed that an image has beentaken in the portrait orientation upon rotating the camera 900 in theclockwise direction. The upper side of the image sensing area in FIG. 2Acorresponds to the right side of the image sensing area in FIG. 3A.Further, in a manner similar to that of FIGS. 2A to 2D, it is assumedthat what is imaged is a subject area in which information “A” to “S”and “a” to “s” enclosed by the dashed line comprises a certain plane,and the zone enclosed by the solid line indicates an observation area asseen from the camera finder. In the portrait orientation, the letterinformation “A, I, Q, H, P” and “d, l, c, k, s” lies outside the imagesensing area and is not imaged.

The image data representing the sensed image is separated into the R, G,and B color components by a series of processing operations in theoriginal image processor 141 in the order in which the image data isread out of the image sensing device, and the data is stored in thefirst memory 142. One apparent horizontal line of the image sensing areashown in FIG. 3A is, e.g., “B, C, D, E, F, G” in case of the first line.In actuality, however, data is read from the line corresponding to theupper side of the image sensing area of FIG. 2A and therefore what isobtained is “G, O, . . . , b, j, r”, as illustrated in FIG. 3B. As aresult, the data of each of the apparent horizontal lines in the imagesensing area shown in FIG. 3A has been stored in order in each of thecolumns of first memory 142. Furthermore, each character imaged has beenrotated 90° in the counter-clockwise direction.

FIG. 3C illustrates the order in which 8×8 pixels, which serve as oneunit, are read out of the first memory 142. Read-out is started from the8×8 pixel block whose image content is “B” at the lower left corner ofFIG. 3B, and the image whose content is “r” at the upper right corner ofFIG. 3B is adopted as the 8×8 pixel block read out last. Thus, in unitsof 8×8 pixels, a raster scan is converted to a zigzag scan to read imagedata out of the first memory 142. However, control is exercised in sucha manner that the order of read-out differs from that in the case wherean image is taken in the landscape orientation.

The image data that has been read out of the first memory 142 istransferred to the work register A71 in rotation processor 7. FIG. 3D isa conceptual view illustrating the order of data of 8×8 pixel units thathave been read out of the first memory 142. As indicated in FIG. 3D, theread-out image data of 8×8 pixel units is data that has undergonerotation by 90° in the counter-clockwise direction. Accordingly,processing for rotation by 90° in the clockwise direction is executed bythe data string converter 72 for every 8×8 pixels and then the resultantdata is stored in the work register B73.

Reference will now be had to FIGS. 4 to 6 to describe processing in therotation processor 7 for rotating 8×8 pixels of image data in a casewhere an image was taken in the portrait orientation.

FIG. 4 is a diagram useful in describing processing for storing 8×8pixels of image data, which have been read out of the first memory 142,in the work register A71.

First, image data is read out of the first memory 142 in 8×8 pixel unitsin the order of read-out shown in FIG. 3C. However, the image data isstored in the work register A71 successively in the order of the pixelrows in a state in which it has not yet been subjected to rotationprocessing, as illustrated in FIG. 3D. In FIG. 4, a symbol such as “Δ”has been applied to each pixel in order to make it easier to understandthe order of the array of pixels. Further, H1 to H8 in FIG. 4 indicatethe rows within the 8×8 pixel unit, and n indicates which numbered pixelfrom the left in FIG. 4 is the pixel, in the overall image shown in FIG.3B, to which the pixel at the upper left corner of the block that is the8×8 pixel unit corresponds. For example, in case of the area where theletter information “G” has been imaged in FIG. 3B, n in FIG. 4 is “1”;if the letter information “O” has been imaged in FIG. 3B, then n is “9”.

FIGS. 5A to 5C are diagrams useful in describing rotation processingexecuted by the data string converter 72. As illustrated in FIGS. 5A to5C, H1 to H8 in an 8×8 pixel unit are stored in 1^(st) to 64^(th)storage areas of the work register A71, as described above in connectionwith FIG. 4.

After the data of each of the pixels has been stored in the 1^(st) to64^(th) storage areas of work register A71, first only the data of thepixels at the head of each of the rows H1 to H8 is transferred to thework register B73 from the H8th row to the H1th row, as illustrated inFIG. 5A. More specifically, first the data of the pixel at the head ofthe H8th row is read out and stored in a first register of the workregister B73. Next, the data of the pixel at the head of the H7th row isread out and stored in a second register of the work register B73. Thus,the data is transferred successively up to the data of the pixel at thehead of the H1th row. In other words, data in an (8N+1)th storage areais transferred in order from N=7 to N=0, and the data is stored in thefirst to eighth registers of work register B73.

Similarly, as shown in FIG. 5B, only the data of the pixels at thesecond position from the head of each of the rows H1 to H8 istransferred to the work register B73. In other words, data in an(8N+2)th storage area is transferred in order from N=7 to N=0, and thedata is stored in the 9^(th) to 16^(th) registers of work register B73.

Similar processing is repeated until the data of the pixel at the lastposition of each row is transferred, and only the data of the pixels atthe eighth position from the head of each of the rows H1 to H8 istransferred to the work register B73, as illustrated in FIG. 5C. Inother words, data in an (8N+8)th storage area is transferred in orderfrom N=7 to N=0, and the data is stored in the 57^(th) to 64^(th)registers of work register B73.

FIG. 6 illustrates the state obtained when the image data in workregister A71 has been stored up to the 64^(th) register of work registerB73 in the manner described above. If this is compared with FIG. 4, itwill be understood that processing for rotation by 90° in the clockwisedirection has been completed at this time.

If the image data of 8×8 pixels following rotation is thus arranged inthe work register B73, the image data is then output to the encoder 10,where it is subjected to encoding and compression processingsuccessively and then stored in the second memory 143. At this time theswitch 74 selects the “1” side to select the output data from the workregister B73.

In the processing for when an image has been taken in the portraitorientation as described above, the work register A71 performs the roleof raising the processing speed as a function of a temporary storagememory when data is sent from the first memory 142 to the data stringconverter 72, and the work register B73 performs the role of raising theprocessing speed as a function of a temporary storage memory when datais sent from the data string converter 72 to the encoder 10, as well asthe role of raising the processing speed relating to encoding processingwhen data is sent from the first memory 142.

FIG. 3E illustrates the concept of the arrangement of an 8×8 pixel blockwhen the data has been stored in the second memory 143. The reason forthe wording “concept of the arrangement” when the data has been storedin the second memory 143 is that since encoding and compressionprocessing is executed in the encoder 10, the image data is actuallystored in a form different from this in the second memory 143.

The image data of the 8×8 pixel block thus subjected to rotationprocessing and encoding and compression processing is storedsuccessively in the second memory 143. As a result, “B” that was at thelower left corner at the time of image sensing, as shown in FIG. 3B, nowis at the upper left corner in the second memory 143, as illustrated inFIG. 3E, and “G” that was at the upper left corner in FIG. 3B now is inthe sixth column of the first row in the second memory 143, as shown inFIG. 3E.

Further, “J” that was second from the lower left corner and “K” that wasimmediately above “J” at the time of photography, as shown in FIG. 3B,now are second from the upper right corner and at the upper rightcorner, respectively, in the second memory 143, as shown in FIG. 3E.

Thus, since the image data is stored in the second memory 143 in theorder illustrated in FIG. 3E, the relative positional relationship ofeach of the blocks comprising 8×8 pixels differs from that of the timeof image sensing shown in FIG. 3B.

Further, by attaching attitude information that prevailed at the time ofphotography to the header of the encoded image information in the secondmemory 143, the attitude at the time of capture of the image to bereproduced can be ascertained. As a result, even though the image datais stored as illustrated in FIG. 3E, an image in the portraitorientation having the correct width and height can be reproduced.

It should be noted that the setting of the compression rate, the size ofan image recorded and output data format, etc., which are required inencoding processing, is performed for example by having the PRS 140 sendthe encoder 10 these values, which have been set by the user prior toshooting, at the moment the shutter switch is pressed.

Further, in the first embodiment set forth above, rotation processing isexecuted every 8×8 pixels. However, if it is so arranged that rotationprocessing is executed in prescribed units, then rotation processingbecomes possible without a need for work registers of large size. Byusing a register capacity that corresponds to the interface with theencoder 10, the optimum capacity is employed and processing thatincludes encoding processing can be executed effectively.

FIGS. 7A to 7C illustrate the state of an observed image when an imageis taken, the output image from the image sensing device and the natureof the display on a display unit such as a TV in a case where the imagewas taken with the camera in the portrait orientation according to thefirst embodiment of the invention.

FIG. 7A illustrates the observed image in the image sensing area in acase where the finder is viewed from the camera eyepiece. Here AFP1 toAFP3 indicate three distance measuring points. Focus is adjusted usingdistance measuring point AFP3. FIG. 7B is a conceptual view of imagedata that is output from the image sensing device, and FIG. 7Cillustrates a display image displayed on the display unit such as a TV.

FIGS. 7A to 7C are for a case where an image has been taken holding thecamera in the portrait orientation. Consequently, the image that isoutput from the image sensing device has been turned on its side byrotation through 90° counter-clockwise, as illustrated in FIG. 7B. FIG.7C illustrates the output image obtained after this image has beensubjected to processing for rotating it 90° in the clockwise directionby the method of the first embodiment described above.

Since the screen of a monitor normally is long in the horizontaldirection, the output image after application of 90° rotation processingis displayed after it is subjected to size-reduction processing in sucha manner that the height of the image will agree with the height of themonitor screen. In this case, blank areas appear on the left and rightsides of the monitor screen but these areas may be displayedmonochromatically, by way of example.

In accordance with the first embodiment, as described above, image datathat has been captured in the portrait orientation is compressed andencoded and then recorded in a storage unit after being subjected torotation processing in accordance with the attitude of the camera whenthe image was taken. As a result, processing for rotation at the time ofdisplay and playback is unnecessary and the time needed for processingfor a portrait-oriented display can be shortened.

Further, since a common size is adopted for the unit area of rotationprocessing for rotating image data and unit area for executing encodingprocessing, rotation processing and encoding can be executedsuccessively on a per-unit area basis, the apparatus can be simplifiedand length of processing time from image sensing to encoding can becurtailed. It is also possible to support an increase in number ofpixels employed in the image sensing device.

Second Embodiment

A second embodiment of the present invention will now be described withreference to the drawings.

According to the first embodiment, encoding is executed after rotationprocessing. By contrast, the second embodiment eliminates encodingprocessing after rotation processing.

FIG. 8 illustrates the structure of an image sensing apparatus accordingto the second embodiment. Here the encoder 10 has been eliminated fromthe arrangement of FIG. 1. The structure of this embodiment is identicalwith that of FIG. 1 in other respects and explanation of these identicalcomponents is omitted.

Although this will depend upon the size of the image information that isoutput from the image sensing unit 13, in the second embodiment an imagethat has undergone rotation processing can be sent to the storage unit 9without degradation of the image by eliminating encoding processing.

In accordance with the second embodiment, as described above, image datathat has been captured in the portrait orientation is recorded in astorage unit after being subjected to rotation processing in accordancewith the attitude of the camera at the time of image sensing. As aresult, rotation processing at the time of display and playback isunnecessary and the length of time required for processing for aportrait-oriented display can be shortened. Further, since encodingprocessing is not executed, an image can be stored, reproduced anddisplayed without a decline in image quality.

Third Embodiment

A third embodiment of the present invention will now be described withreference to the drawings.

According to the first embodiment, after image data has been encodedfollowing rotation processing, the encoded data is temporarily stored inmemory, and then output to the storage unit 9. By contrast, in the thirdembodiment, encoding processing is executed after image data has beenstored in memory following rotation processing, and the encoded outputis delivered to the storage unit 9 without being stored temporarily inmemory.

FIG. 9 illustrates the structure of an image processing apparatusaccording to a third embodiment of the invention.

In FIG. 9, the second memory 143 has been eliminated from the structureof FIG. 1 and two FIFO memories have been inserted. Further, theinsertion of the two FIFO memories is accompanied by the addition of twoswitches for selecting either of these FIFO memories. Other structuralcomponents are the same as those of the first embodiment in FIG. 1 andthe explanation of them is omitted. Processing involving the FIFOmemories will now be described.

In a case where an image has been taken in the portrait orientation,rotation processing in which 8×8 pixels are adopted as one unit isexecuted by the rotation processor 7 starting from any of the R, G, andB color components, as described in the first embodiment, and the outputresulting from this processing is input to either one of two FIFOmemories 302, 303 via a switch 301.

When storage of 8×8 pixels of data is completed, the switch 301 changesover the output destination to the other one of the FIFO memories. Whenthe next cycle of rotation processing is executed, the resulting outputis stored in the FIFO memory at the destination to which the changeoverhas been made and the FIFO memory in which storage of data has alreadybeen completed outputs its content to the encoder 10 successively viathe switch 304. The image data that has been input to the encoder 10 isencoded and then output to the storage unit 9.

The FIFO memories 302, 303 each have a capacity equivalent to 64 pixels(8×8 pixels). The two FIFO memories 302, 303 are changed overalternatingly by the switches 301, 302.

Further, since it is necessary that the processing speeds per pixel fromthe first memory 142 to the encoder 10 be made the same, processing isexecuted at a processing speed governed by whichever block has thelowest speed. However, since the memory that delivers the data from therotation processor 7 to the encoder 10 is a FIFO memory, a cost-relatedadvantage is obtained.

In accordance with the third embodiment, as described above, image datathat has been captured in the portrait orientation is compressed andencoded and recorded in a storage unit after being subjected to rotationprocessing in accordance with the attitude of the camera at the time ofphotography. As a result, rotation processing at the time of display andplayback is unnecessary and the length of time required for processingfor a portrait-oriented display can be shortened.

It should be noted that in the first to third embodiments, the signalthat is to undergo rotation processing need not be one having R, G, andB color components and may be any signal with which the functions of thepresent invention can be achieved.

Further, application of the present invention is not limited to adigital camera. The invention can be applied to optical equipment anddevices other than cameras and is applicable to any device in which animage signal obtained by image sensing using photoelectric conversionelements arrayed in two dimensions is converted and finally output to anoutput unit such as a monitor or printer.

Further, it would be obvious to those skilled in the art that it is alsopossible to readily support rotation processing in a case where an imagehas been rotated by 180° in the clockwise direction or by 90° in thecounter-clockwise direction and not only 90° in the clockwise direction.

Further, the embodiments set forth above deal with a hypothetical casewhere rotation processing is executed in such a manner that an imagethat has been captured in the portrait orientation is made to conform toa landscape-oriented monitor screen. However, in a case where themonitor screen of the display unit used in reproducing the image is longin the vertical direction, it will suffice if similar rotationprocessing is applied to an image that has been captured in thelandscape orientation. Accordingly, it will suffice to set beforehandhow rotation is to be performed when the image sensing apparatus is inany particular attitude in accordance with a reference orientation ofthe image sensing apparatus and the desired shape of the monitor screen(e.g., depending upon whether the monitor screen of a display unitmainly used to observe an image shot by the user is long horizontally orlong vertically).

Other Embodiments

The present invention can be applied to an apparatus comprising systemconstituted by a plurality of devices (for example, host computer,interface, and camera head) or to a single device (for example, digitalstill camera, digital video camera).

Furthermore, the invention can be implemented by supplying a softwareprogram, which implements the functions of the foregoing embodiments,directly or indirectly to a system or apparatus, reading the suppliedprogram code with a computer of the system or apparatus, and thenexecuting the program code. In this case, so long as the system orapparatus has the functions of the program, the mode of implementationneed not rely upon a program.

Accordingly, since the functions of the present invention areimplemented by computer, the program code installed in the computer alsoimplements the present invention. In other words, the claims of thepresent invention also cover a computer program for the purpose ofimplementing the functions of the present invention.

In this case, so long as the system or apparatus has the functions ofthe program, the program may be executed in any form, such as an objectcode, a program executed by an interpreter, or scrip data supplied to anoperating system.

Example of storage media that can be used for supplying the program area floppy disk, a hard disk, an optical disk, a magneto-optical disk, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, a non-volatile type memorycard, a ROM, and a DVD (DVD-ROM and a DVD-R).

As for the method of supplying the program, a client computer can beconnected to a website on the Internet using a browser of the clientcomputer, and the computer program of the present invention or anautomatically-installable compressed file of the program can bedownloaded to a recording medium such as a hard disk. Further, theprogram of the present invention can be supplied by dividing the programcode constituting the program into a plurality of files and downloadingthe files from different websites. In other words, a WWW (World WideWeb) server that downloads, to multiple users, the program files thatimplement the functions of the present invention by computer is alsocovered by the claims of the present invention.

It is also possible to encrypt and store the program of the presentinvention on a storage medium such as a CD-ROM, distribute the storagemedium to users, allow users who meet certain requirements to downloaddecryption key information from a website via the Internet, and allowthese users to decrypt the encrypted program by using the keyinformation, whereby the program is installed in the user computer.

Besides the cases where the aforementioned functions according to theembodiments are implemented by executing the read program by computer,an operating system or the like running on the computer may perform allor a part of the actual processing so that the functions of theforegoing embodiments can be implemented by this processing.

Furthermore, after the program read from the storage medium is writtento a function expansion board inserted into the computer or to a memoryprovided in a function expansion unit connected to the computer, a CPUor the like mounted on the function expansion board or functionexpansion unit performs all or a part of the actual processing so thatthe functions of the foregoing embodiments can be implemented by thisprocessing.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

CLAIM OF PRIORITY

This application claims priority from Japanese Patent Application No.2004-228014, filed on Aug. 4, 2004, which is hereby incorporated hereinby reference herein.

1. An image processing apparatus for processing image data that has beencaptured by an image sensing apparatus having a two-dimensional imagesensing device, said apparatus comprising: an attitude information inputunit that inputs attitude information representing the attitude of theimage sensing apparatus; a rotating unit which divides the image datacaptured by the image sensing device into a plurality of divided imagedata that have a block size corresponding to a size by which aprocessing unit processes image data, and rotates the block-size imagedata in accordance with the attitude information input by said attitudeinformation input unit; and a processing unit that processes the imagedata output from said rotating unit in units having the block size;wherein in a case where the attitude information indicates that theimage sensing apparatus is not rotated relative to a reference attitude,said rotating unit inputs the block-size image data successively in afirst order and outputs the block-size image data without rotating theblock-size image data, and wherein in a case where the attitudeinformation indicates that the image sensing apparatus is rotatedclockwise through at least any one of 90°, 180° and 270° relative to thereference attitude, said rotating unit inputs the block-size image datasuccessively in a second order which is determined by the relationbetween the first order and the rotation degree of the image sensingapparatus, rotates each of the block-size image data, and outputs theplurality of the rotated block-size image data.
 2. The apparatusaccording to claim 1, further comprising a memory that stores one frameof image data that is output from the image sensing apparatus; whereinin a case where the attitude information indicates the rotation degreeis 90° or 270°, said rotating unit reads out image data, which has beenstored in said memory, in order by the blocks, in an orthogonaldirection to a line direction of the image data, and applies rotationprocessing to the image data in units of blocks.
 3. The apparatusaccording to claim 1, wherein said rotating unit includes: a firststorage unit that has a capacity, which is equivalent to the block size,for storing the divided image data before the rotation processing isexecuted; and a second storage unit that has a capacity, which isequivalent to the block size, for storing the divided image data afterthe rotation processing has been applied thereto.
 4. The apparatusaccording to claim 1, wherein said processing unit is an encoding unitthat successively encodes the image data, which has been processed bysaid rotating unit, in units having the block size.
 5. The apparatusaccording to claim 4, further comprising a storage unit that storesimage data that has been encoded by said encoding unit.
 6. The apparatusaccording to claim 1, further comprising a storage unit thatsuccessively stores image data, which has been processed by saidrotating unit, on a storage medium in the units of the block size.
 7. Animage sensing apparatus having a two-dimensional image sensing device,comprising: an attitude detecting unit that detects the attitude of theimage sensing apparatus; a rotating unit which divides the image datacaptured by the image sensing device into a plurality of divided imagedata that have a block size corresponding to a size by which aprocessing unit processes image data, and rotates the block-size imagedata in accordance with the attitude information input by said attitudedetecting unit; and a processing unit that processes the image dataoutput from said rotating unit in units having the block size; whereinin a case where the attitude information indicates that the imagesensing apparatus is not rotated relative to a reference attitude, saidrotating unit inputs the block-size image data successively in a firstorder and output the block-size image data without rotating the blocksize image data, and wherein in a case where the attitude informationindicates that the image sensing apparatus is rotated clockwise throughat least any one of 90°, 180° and 270° relative to the referenceattitude, said rotating unit inputs the block-size image datasuccessively in a second order which is determined by the relationbetween the first order and the rotation degree of the image sensingapparatus, rotates each of the block-size image data, and outputs theplurality of the rotated block-size image data.
 8. An image processingmethod for processing image data that has been captured by an imagesensing apparatus having a two-dimensional image sensing device, saidmethod comprising: inputting attitude information representing theattitude of the image sensing apparatus; in a case where the attitudeinformation indicates that the image sensing apparatus is not rotatedrelative to a reference attitude dividing the image data captured by theimage sensing apparatus into a plurality of divided image data having ablock size corresponding to a size by which processing is performed,inputting the block-size image data successively in a first order, andoutputting the block-size image data without rotating the block-sizeimage data; and in a case where the attitude information indicates thatthe image sensing apparatus is rotated clockwise through at least anyone of 90°, 180° and 270° relative to the reference attitude dividingthe image data captured by the image sensing apparatus into a pluralityof divided image data that have a block size corresponding to a size bywhich processing is performed; inputting the block-size image datasuccessively in a second order which is determined by the relationbetween the first order and the rotation degree of the image sensingapparatus, and rotating the block-size image data in accordance with theattitude information input in said attitude information input step; andprocessing the rotated image data in units having the block size.
 9. Acomputer-readable recording medium on which has been recorded a programfor causing the image processing method set forth in claim 8 to beexecuted by a computer.